HPC Crash Course

Session Outline

• Introduction to HPC

Attempt at a Definition: HPC …

HPC in Biomedical Research

Trade-Offs of HPC

There is no free lunch!

What is Your Take? 🤸

So far

• have you benefited from advantages?
• have you suffered from drawbacks?

From here on

• what do you hope to gain from using HPC?
• what risks do you see?

There Is No Magic (1)

There Is No Magic (2)

There Is No Magic (3)

• Is it easy to program an iPhone?
  • “yes and no”
• Is it easy to program for an HPC?
  • “yes and no”

“Es gibt keinen Königsweg” (1)

“Es gibt keinen Königsweg” (2)

• Master using Linux.
• Master programming for
  • a single core
  • multiple cores on a single machine
  • multiple cores on multiple machines
  • programming GPUs
• Realize that most problems
  • are embarassingly parallel

“Es gibt keinen Königsweg” (3)

• Master using Linux.
• Master programming for
  • a single core
  • multiple cores on a single machine
  • multiple cores on multiple machines
  • programming GPUs
• Realize that most problems
  • are embarassingly parallel
  • have well-solved “cores”

Types of Parallelism

Single-core level (not in focus here)

• Bit-level parallelism
  • aka bit-wise AND, OR, etc.
• Instruction-level parallelism
  • aka instruction pipelining

Programming level

• Pipeline parallelism
• Task parallelism
• Data parallelism

Pipeline Parallelism

Task Parallelism (1)

Task Parallelism (2)

Compute Nodes (1)

“Same-same (as your laptop), but different.”

Compute Nodes (2)

More differences from “consumer-grade” hardware:

Your Experience? 🤸

• have you used shared/distributed
  memory parallelism before?
• what is your experience/hope?

Cluster Architecture

(Distributed) File Systems

Distributed File Systems (2)

“Same-same (as your laptop), but different.”

• POSIX file system
  • laptop: ext4/XFS/btrfs/ZFS/…
  • distributed: CephFS, GPFS/SpectrumScale, BeeGFS, …
• POSIX guarantees
  • sync() → visible everywhere
  • mkdir/open() → visible everywhere
  • files can be opened by multiple processes
• … harder to enforce in a distributed (multi-node) setting

Distributed File Systems (3)

Best Practices / Do’s & Don’ts

• use modest sized directories (<10k entries)
  • don’t create one file per gene (or similar)
  • create sub directories, e.g., abcdef → ab/cdef
• don’t splurge in file count
  • don’t create one file per NGS read (or similar)
• avoid recursive traversal of large structures
  • ls -lR will be slow!

Distributed File Systems (4)

Best Practices / Do’s & Don’ts

• avoid small reads/writes and random access
  • each I/O operation (IOP) needs to go through the network
  • I/O systems are better at handling larger/sequential reads/writes
• DO stream through your files
  • for each line/record in file: # do work
• DO use Unix sort
• DO use Unix pipelines rather than temporary files
  • e.g., seqtk mergepe R1.fq R2.fq | bwa mem ref.fa | samtools sort | samtools view -O out.bam

Distributed File Systems (6)

Best Practices / Do’s & Don’ts

• DON’T be afraid of large files
  • 1GB does not qualify as large
  • 1TB files … do

Job Scheduler and Resource Management

sequenceDiagram
    autonumber
    User-)+Scheduler: sbatch $resource_args jobscript.sh
    Scheduler->>+Scheduler: add job to queue
    User-)+Scheduler: squeue / scontrol show job
    Scheduler-->>+User: job status
    Note right of Scheduler: scheduler loop
    Scheduler-)Compute Node: start job
    Compute Node->>Compute Node: execute job
    Compute Node-)+Scheduler: job complete

Connecting to the HPC

• Windows Users
  • Should follow the hpc-docs instructions
• Linux / Mac Users
  • Should be able to do `ssh -l $USER_{c,m} hpc-login-1.cubi.bihealth.org

Introduction

• Python has a threading library with low-level primitives
• But why is there a multiprocessing? Processes?
  • Python has a global interpreter lock (GIL)
  • (maybe removed at some point but it is there)
  • Access to Python data structures is serialized
  • Multithreading: only while threads are blocked (I/O)
• Multiprocessing to the rescue:
  • Copy data to another process
  • This process can do the work

Reminder: apply/map

• Remember functional programming?
• map(func, list) -> list
  • Apply func to each element on the list to obtain a new list of same size
  • Can be parallelized if there are no side effects
• apply(func, list)
  • Apply func to each element on the list, ignoring results
  • Can be parallelized if there are no side effects

Thread Pools

• Thread pools are abstractions for parallelism
• We create a pool with N threads
• We let the thread pool process collections / lists of work
  • multiprocessing.Pool() (process pool ;-))

Example

import multiprocessing

def func(element):
    # ...

if __name__ == "__main__":
  work = list(range(1_000_000))

  with multiprocessing.Pool(processes=4) as p:
    # force single element chunks
    done_work = p.map(func, work, chunksize=1)

  print(done_work)

Caveats

def func(element, a, b):
  # ...

def func2(element_params):
  element, params = element_params
  func(element, **params)

# ...

params = {
  "a": 1,
  "b": 2,
}

real_work = list(range(1_000_000))
work_with_params = [(x, params) for x in real_work]
done_work = p.map(func, work, chunksize=1)

Introduction

• GNU parallel is a command line tool that allows you to
  • run a potentially large list of tasks
  • with a fixed number of parallel tasks at the same time
• It is similar to the earlier Python multiprocessing.ThreadPool
• Excellent Tutorial

Example

THREADS=4

parallel -t -j $THREADS \
  'md5sum {} >{}.md5' ::: *.bam

parallel -t -j $THREADS \
  'cd {//}; md5sum {/} >{/}.md5' ::: *.bam

File System Quotas

• Your home volume has tight quotas

• Some programs will write a lot there

• Solution: use work volume

  NAME=.cpan
  mkdir -p work/$NAME
  mv $NAME/* work/$NAME
  rmdir $NAME
  ln -sr work/$NAME $NAME

• E.g., run the above for NAME in ondemand miniconda3 R Downloads .apptainer .theano .singularity .npm .nextflow .local .debug .cpan .cache .aspera

Killed Out Of Memory Jobs

• If you get random Killed messages, your job probably needs more memory than available
• In particular:
  • On login nodes, you only have 100MB of RAM!
• Solution: request more memory!
  • More on this in the Slurm session